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ABSTRACT 

The purpose of the audit report on the Bilingual 
Education Program (1969-70) of the Harlandale and San Marcos 
Independent School Districts is to give a critique on the quality of 
the project evaluation, to discuss the comparative findings of the 
project evaluation and the audit, and to confirm or question the need 
for program modifications proposed in the evaluation. . The audit 
report is divided into 5 sections: (1) introductory and general 
comments on the quality of the project evaluation and the comparative 
findings of the project evaluation and the audit; (2) detailed 
critique of the product, process, and management evaluation for each 
component, based cn an assess .ent of the instruments used, data 
collection procedtires^ data analysis techniques, and data analysis 
presentation; (3) description of the auditor^s on-site visit findings 
and correlation with the evaluator's data and reports on a component 
by component basis, summary of consistencies and discrepancies; (4) 
recommendations for evaluation design revisions with a rationale for 
each recommendation; and (5) confirmation or questioning of the need 
for program modifications proposed in the project evaluation. 
(Author/NQ) 
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EDUCATIONAL ACCOMPLISHtiENT AUDIT REPORT 



♦ Purpose of the Report 

This Audit Report is made in compliance with the contractual agree- 
ments, legal prescriptions, and official direjef^ves under the Provisions 
of Title VII of Public Law 89-10, as aniGaded, for the establishment and 
operation of bilingual education programs. This is an Audit Report on 
the Bilingual Education Program of the Harlandale Independent School 
District, The San Marcos Independent School District, and the Southwest 
Texas State University, with Dr. E. Hughes as program director* 
The purpose of th.ls report is to give a critique concerning the quality 
of the project evaluation and discuss the comparative findings of the 
project evaluation and the audit and to confirm or question the need for 
program modifications which have been proposed as a result of project 
evaluation. 

Scope of the Report 

^ The Audit Report follows rather closely the Suggested Audit Report 

Content Areas received from the United States Office of Education and 
made a part of the contractual agreement between the auditor and the 
program director, as follows: 

1. Introductory and general comments concerning the quality 
of the project evaluation and the comparative findings of the 
project evaluati^rn and the audit. 

2. Detailed critique of the product, process, and management 
evaluation conducted for each component, based on an assess- 
ment of the instruments used, data collection procedures, data 
analysis techniques, and data analysis presentation. 
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3. Description of the auditor's on-site visit findings and 
their correlation with the evaluator's data and reports, on a 
component by component basis; summary of consistencies and 
discrepancies, and interpretation of the discrepancies. 

4. Recommendations for revisions of the evaluation design, 
including a rationale for each recommendation. Since the 
auditor's objectivity can be retained only if the selection 

of a specific corrective action is a local dedision, he should 
provide general rather than specific recommendations, posing 
several alternative actions or possibel sources of assistance 
to the LEA. in correcting the deficiency. 

5. Confirmation or questioning of the need for program modi- 
fications which have been proposed as a result ot project 
evaluation 

Each of the audit areas described above will be referred to as sections 
with the section numbers corresponding with the audit area description 
nunbers . 

SECTION I 

General Comments Concerning, the Quality 
of the Project Evaluation and on 
Comparative Findings 

The extent of the effectiveness of an evaluation is primarily 
determined by how well the evaluating techniques quantify through objec- 
tive measurements the quality and extent of pupil learning for which 
the project is chiefly responsible. A careful examination of the 
written evaluation, as well as the evaluation design, techniques, and 
procedures, reveals that this has been its major objective — to 
measure the learning outcomes and the progress in learning. These ^ 
measurements have been in terms of the extent of attainment of behav- 
ioral objectives and in comparisons with control groups and baseline 



data. Some of these measures have been obtained by well recognized and 
properly validated tests using standard procedures. Some have been * 
obtained by rather hastily contrived scales and tests that appear to 
have conslderaj/le merit and face validity. Other local tests were to 
measure the learning achievement In terras of the specifics of the locally 
designed curriculum program and were most ambitiously conceived but were 
not completed In time to be tried out and validated for use In spite of 
the diligent work of the teacher writing teams under coordinator direc- 
tion. The Intervals between pre- and post-tests were In many instances 
much too short for significant changes in quantity of learning to have 
taken place. 

In general the evaluation has much merit and is v/ell done. The 
evaluation design, procedures, and techniques, have emerged as the pro- 
gram developed. This could not have been otherwise since the directives 
from the United States Office of Education were not really clarified 
until after the Evaluation Conference at Newport Beach, California, in 
November. Considerable revision was necessary to implement the princi- 
ples set forth in that conference and subsequent interpretative commun- 
ications. A reformulation of the behavioral objectives shortened some 
of the pre- and post- test intervals and delayed local test construction. 

' The shortcomings which have just been mentioned should not lead to 
the hasty inference that the evaluation is inadequate. The evaluation 
has a remarkably strong base and is most adequate. This is especially 
true when one considers that the developing program was so slow in 
assuming a measurable structure. 
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The evaluation places heavy emphasis on whatever interpretations 
may be obtained by examining and comparing the size of the means of the 
scores. As important a. this is, the meanings and interpretations that 
might have been obtained from the other measures of variability and 
relative positions for individual scores are not represented adequately 
by the means. This leaves unanswered questions about how the various 
aspects of the program affect the slower than average or the faster than 
average learners. The extreme differences among the sections ore not 
adequately accounted for by a comparison of means. Again, in all fair- 
ness, it must be noted that there are very real extenuating circumstances. 
The Project Director is only half time. Not only does he have total 
responsibility for project management and direction but he has to do the 
evaluation, too. In addition to the fact that there was inadequate 
time to do the most thorough type of evaluation, there is some question 
as to whether anyone should be called on to do the principal evaluation 
of his own handiwork. The project should have an evaluator who could 
devote at least fifty percent of his time to evaluation. 

In spite of all that has been said in this analysis, the evalua- 
tion is adequate for a first year, and has much merit. The findings 
are most credible. 

The differences in the findings of the evaluator and the auditor 
were miniscule. Whatever differences that do exist are more of an inter- 
pretative nature than of fact. They do not merit mention in this gen- 
eralized discussion, for they arc not worthy of that much emphasis. 
Those that do exist will be specifically pinpointed in subsequent sec- 
tions in their proper context. 
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, ^ SECTION II 

Critique of the Product, Process, and 
Management Evaluation for Each Component 

1. Curriculum Development Component , The development of 
curriculum outlines, teaching units, and testing devices, and 
assembling of various types of teaching materials suitable 
for bilingual instruction in grades 1-3 c 

The pertinent facts in the evaluation concerning this component, 
briefly summarized, are somewhat as follows: Harlandale had a rudimen- 
tary bilingual guide. This and other curriculum and audio-visual mate- 
rials were examined. Twp writers from each of the two districts, work- 
ing with an instructional systems consultant, produced curriculum 
guides in the summer of 1969. * 

The indicated evaluation of these guides consisted of their 
approval by the director, the coordinators, by college professors at 
Southwest Texas State University, by curriculum supervisors, and by 
consultants. The use of the expert jury technique for the preliminary 
evaluation of the curriculum guides is justified since there is no 
adequate way of evaluating curriculum materials until they are empir- 
ically tested in the classrooms. 

During the year the teachers who used the materials made comments 
and suggestions to be used by the writing teams in making revisions 
during the summer of 1970. The evaluation report could have Included a 
sampling of the kinds and types of suggestions that the teachers made 
for the revisions. It would also be interesting to know the extent to 
which the teachers participated in making these revision suggestions. 
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In response to questions, a few of the- teachers made very specific sug- 
gestions to the auditor, but in most Instances the comments were gen- 
exallzed and indicated approval of the materials • This component is so 
Interlocking with the instruction component that the real evaluation of 
materials comes in their effectiveness in improving the instructional 
program through increasing the amount of learning. 

The locally made tests to accompany the curriculum guides were not 
completed in time to be reproduced and tried out. The auditor is aware 
that considerable effort has been put forth in this enterprise but 
neither the tests nor what they purport to measure can be evaluated at 
this time. 

2. Instructional Component . Instruction in both Spanish and 
English. 

The evaluation of this component begins with the point that both 
the teachers and pupila were willing participants in the program and 
that the parents and the communities strongly endorsed the bilingual 
approach to instruction. This point is well established throughout the 
evaluation and is confirmed by the auditor's on-site visits. 

The schools, the teachers, and the number of pupils in each class 
are reported in Table I. The mean number of pupils was found to be 
27.7. The further observation might be made that the v2?riation from 
this mean was slight with a low of 24 and a high of 32 in the public 
schools. St. Johns has a low of 23. It seems to be rather well 
established that 27 pupile are too many for the most effective instruc- 
tion in the first gra^^e. The number of pupils who spoke little or no 
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English is surprisingly low, varying from 0 to 29 in the first grades. 
However, these figures should not be taken too seriously, for the 
auditor has found that teachers vary greatly in their judgment of 
what constitutes little or no English. 

It might be useful to make a more careful and detailed study of 
the demographic characteristics of the pupil population in each of 
the participating elementary schools. These factors could affect the 
curriculum and the instructional program needed in the schools. 

The evaluator has commented on the problem with having usable . 
control groups. It almost impossible to have effective control 
groups in the pubJUc schools because administrative convenience and 
the efforts to keep teacher-pupils loads equalized makes it difficult 
to keep them constant or equal with the experimental groups. But 
even with the unfavorable circumstances that exist some useful com- 
parisons can be laade. While it is true that control groups were 
initially higher than the bilingual groups in mean I. Q. as measured 
by the Otis Alpha Mental Ability Test, Non-verbal, it should be not 
that the bilingual sections achieved more learning and ability progress 
during the school year than the control groups did. Since the I. Q. 
as reported in Table II of the evaluation reflects the effect of a 
moving chronological age in the pre- and post-tests of the Otis, to 
maintain the same I. Q. score is to indicate normal progress in learn- 
ing with the general population of first grade pupils. This certainly 
is a most commendable achievement for a bilingual^ group. In fact a 
further analysis of the scores shows that 16 of the 20 first grade 



bilingual sections made some gain in nean I* scores in the post- 
test • Administration of the verbal portion of the Otis Test along 
with th^ non-verbal in the post- test sessions would have probably 
given more credibility to the findings from comparative !• scores. 
The use ox^ther statistical measures along with the mean would have 
made further analysis possible, particularly the progress of high 
and low pupils in the first grade sections. It would also be helpful 
to know how well the Goodenough Draw-a-Man.Tcst , the Otis Alphs Men- 
tal Ability Test, Non-verbal, and the Metropolitan Readiness Test 
correlate with one another when administered bilingually with a pupil 
population that has a high percentage who have Spanish as a first 
language and live in low-income areas. If" the correlation is high 
among the three tests then perhaps it would uot be necessary to give 
all three tests. Much additional information could be extracted from 
many of the measures by an evaluator who had more time and was not 
encumbered with directorship duties. 

The Metropolitan Reading Readiness Tests and the Metropolitan 
Achievement Test, Primary I Battery, v;ere administered in October and 
May respectively, and the scores mqvc treated as pre- and post-test 
data for the measurement of learning in language arts and mathematics. 
Table III and Table IX in the evaluation. Even though there is some 
rationale for this procedure and a belief that the results V70uld be 
loosely indicative of the extent of the learning in these areas, it 
would be entirely erroneous to accept at face value the unfavorable 
results portrayed by the mathematical differences between the pre- and 
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post-tests. The Me::ropolltan Readiness Test required no reading abil- 
ity and was given blllngually The Metropolitan Achievement Test, 
Primary Battery I, required reading and was given In English to obtain 
both the reading and mathematics scores* To expect these percentile 
scores to be equivalent Is unreasonable. These statements are further 
substantiated by the unfavorable comparison with the control groups 
which had a higher percentage of English speaking children. It is not 
the purpose of this program to compensate totally for the language 
deficiency in one year. It is not even surprising that the control 
groups lost 18 percentage points while the bilingual groups were losing 
23 percentage points. It suggests that even the control groups prob- 
ably have enough Spanish speaking pupils to profit from bilingual 
instruction. Furthermore, it is not surprising that the Oral Language 
Education group (Thompson) was the lowest in the two school systems. 
'Concepts, not excessive drill, are needed here. The methods of measur- 
ing the learning in transition from readiness activities to actual 
reading should be reviewed. 

The Metropolitan Achievement Test scores in reading are much more 
valid on a pre- and post-test basis and show normal progress with the 
genera] pupil population at the second and third grade levels in that 
the means of the percentages do not deviate from constancy appreciably. 
The second and third grade pupils achieved this even though most of 
them had not previously been taught blllngually. Much more data could 
be obtained for analysis by looking beyond the general means. Com- 
parison of grade level equivalents analyzed by sections could be quite 



reveallnr,. Relating this to the amount of time spent on Spanish might 

prove more interesting. 

lo the absence of any satisfactory tests in Spanish language pro- 
ficiency, the program staff and the teachers were quite ingenious in 
creating interim tests. These tests apparently do measure the ability 
to pronounce Spanish words, to associate certain words with obj-acts, 
to spell in Spanish, and to make statements in Spanish. This measures 
to some extent familiarity with the Spanish language and seme aspects 
of its usage. However, since Spanish is a highly phonetic language, 
it is possible for a pupil to .ead orally with very little comprehension 
or conceptual understanding. Tlic auditor agrees with the evaluator 
that the search for suitable Spanish language tests or the development 
of such tests must be pursued with much vigor. The use of the Pruebr-s 
Lectura Interamericano Scrie at all grade levels except the first on a 
pre- and post-test basis is advisable if better tests cannot be found. 
Even in the first grade it should be given at the end of the year. 

It should not be overlooked th.'>t the Spani.-h testing was done in 
accordance with the behavioral objectives and indicated goal achieve- 
ment except in spelling. 

Coramants have already been made on the first grade mathematics 
test. The evaluator's analysis and comments on the second and third 
grade mathematics tests say all that need to be said. 

The development of social concepts and skills in relation to the 
Mexican-American culture and heritage was measured only through class- 
room tests given by the individual teacher. The teachers report that 
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progress in these phases of the social studies program was satisfactory 
but no data are available. More adequate measure of the learning in the 
social studies awaits the completion of the locally made tests. 

The efforts to measure the development of the self concept were com- 
mendable and may be quite effective when implemented as the evaluator 
has recommended. The auditor recommends that the use of attendance as 
any kind of a measure of self concept be discontinued. This measure is 
not compatible with the culture. 

3. Teacher Improvement Component. The development of a com- 
prehensive program of teacher improvement through systematic 
preservice and inscrvice training of teachers. 

The evalua tor's report of the preservice teacher training aspects 
of this component seem to be entirely accurate and the program officials 
are to be commended for this contribution to bilingual education. The 
teachers in both Harlandale and San Marcos were high in their praise of 
the efficient help and contributions made by these student teachers and 
internes. The teacher aides were also a definite asset to the program. 

The auditor was able to attend quite a few of the inservice train- 
ing sessions and found them to be quite effective. Perhaps the fre- 
quency of these sessions could be increased and their scop<a made more 
comprehensive. The teachers seemed to profit from the sessions held if 
one may judge by teacher comments. 

The use of the teacher rating scales is cononendable. The coordina- 
tors and the principals used discriminating judgment. It is the 
auditor's opinion that the teachers were generally rated a little too 
high even though there is no question that they are doing a highly * 
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satisfactory job of teaching as a whole. The degree of^eacher satis 



faction speaks well for the program. It is obvious that the teacher 
aides need a continuous and specific training program. 

The data on teacher ratings reported In Table XVII, XVIII, and 
XIX would be more revealing if they were not all combined into one com- 
posite score for each teacher. 

4. Community Involvement Component . The development of 
•appropriate strategies fcr maximizing the involvement of 
parents and community resources. 

As previously Indicated all available evidence Indicates that both 
communities had strong parental and commur'':y endorsement of the biling- 
ual instructional program. The auditor joins the evaluator in commenda- 
tion of the Harlandale coordinator on the success of the Parents' Circle 
meetings. The high percentage of questionnaires to parents returned 
further reflects the success of these meetings and other efforts to 
Involve the parents. Perhaps these meetings can be held more consist- 
ently and more regularly at all the schools. 

5. Project Coordination Component s The development of an 
efficient, cooperative administrative structure among the 
schools . 

This component is so interlocking with the other components that 
it can hardly be considered a separate component. Since this program 
Is in the nature of an administrative overlay rather than completely 
Integrated into the administrative organization of the two school 
districts, naturally there have been problems where the overlay crinkled 
a little bit at times « The auditor has observed that there was always 
an overwhelming desire on the part of all persons concerned with major 




responsibilities to be cooperative with perhaps one or two temporary 
exceptioijs. The project director has been especially successful in 
welding together the many aspects of the program and many persons 
involved into a productive ongoing bilingual instructional program. 
In this he has had the efficient cooperation of the Harlandale coordi- 
nator who is held in highest esteem by the Harlandale teachers both as 
a person and as a bilingual educator. The principals and top administra- 
tive officials have been most supportive. For a consortium with the 
distances involved, it has done extremely well in coordinating its 
activities. In fact, it may have done this too well with curriculum 
materials with not enough allowance for the individual differetlces 
between the districts. The auditor found the teachers m©st cooperative 
in both' districts. 

SECTION III 

Description of the Auditor's On-site Visit 
Findings and their correlation with the 
Evaluator's Data and Reports on a Com- 
ponent by Component Basis; Summary of 
Consistencies and Discrepancies, and 
Interpretation of Discrepancies 

The auditor's on-site visits consisted of visiting classrooms and 
observing teaching and learning activities, talking with teachers to 
obtain their viewpoints and attitudes, discussions with the principals, 
frequent discussions with the program coordinators, particularly in 
Harlandale, many visits with the project director, discussions with 
school administrators in both districts, examination of curriculum 
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guides and teaching materials in the classrooms and in the central 
offices, examination of tests and proposed tests, looking at test scores 
and test results, and talking with teacher trainees working in the pro- 
ject. 

The scope of this section overlaps with the previous section deal- 
ing with the critique so much that many of the things called for in this 
section have already been discussed. However, some of them will be 
reviewed and reemphasized here. 

1 . Curriculum Development Component . 

There were no real discrepancies in the findings of the evaJuator 
and the auditor in this component. However, questions were raised. 

(1) Are the pupil populations of San Marcos and Harlandale so similar 
that their curriculum materials can be prepared by a unified writing 
team and the two districts use identical or similar curriculum materials? 

(2) To what extent did the classroom teachers actually make written sug- 
gestions for the revision of the curriculum materials? 

2. Instructional Component : 

The evaluator and the auditor both find that the teachers and 
the pupils are willing participants in the program and that the pro- 
gram has a strong parental and community endorsement in both school 
districts. 

The auditor suggests that the demographic characteristics of the 
pupil populations be explored in more specific detail in the future to 
ascertain curricular and instructional needs. 
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The evaluator and the auditor arc in substantial agreement on the 
meaning and Interpretation of the test results. However, the auditor 
has suggested that additional information might be obtained from the 
data by using other statistics in addition to the mean so that the 
learning progress of the high and low scoring pupils, those who deviate 
somewhat from the mean, might be more fully explored. It is also 
observed that this cannot be done very satisfactorily until an evaluator 
is added to the project, for the program director, who is now the 
evaluator, docs not have time to go into this much statistical detail. 

The auditor questions whether the Metropolitan Readiness Test and 
the Metropolitan Achievement Test, Primary Battery I, should be used in 
pre- and post- test sequence for measuring growth in language arts and 
mathematics in the first grade. The evaluator also indicaired that this 
should be reviewed. 

Comments on the inadequacy of the Spanish language tests were dis-- 
cussed in considerable detail in th^j previous sections. The auditor 
recommends using Pruebas Lectura Interamericano Serie if better tests 
are not found. 

The auditor is in agreement with evaluator comments and 
analysis of the second and third grade mathematics tests. 

The auditor shares with the evaluator the opinion that the tests 
made by individual classroom teachers are somevjhat less than adequate 
in social studies and looks forward to the ciore comprehensive tests 
now in preparation. The auditor and evaluator are in agreement that 
McDanlel^s Inferred Self-concept Scale may be useful in measuring 
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self-concept development, but the auditor recommends that the use of 
attendance as an implied measure of self-concept be discontinued since 
this measure is not compatible with the culture • 

3 . Teacher Improvement Component . 

The findings and observations of the auditor and the evaluator in 
this component are in almost total correlation. 

The auditor recommends a more detailed analysis of the teacher 
rating scales rather just reporting the composite score. 

4 . Community Involvement Component * 

The auditor joins the evaluator in commending the Harlandale 
coordinator in the use of the Parents' Circle for community and parental 
involvement and recommends that it be expanded. 

5 . Project Coordination Component . 

The program administrators are commended for welding many diverse 
persons and interests, even geographical separation, into a coordinated, 
effective learning program and reducing contradirectional forces to a 
minimum. 

SECTION IV 

Recommendations for Revisions of the 
Evaluation Design 

The recommendations for revisions of the evaluation design were 
all made in direct context relationships in Section II, the Critique 
of the Product, Process, and Management Evaluation of Each Program 
Component. Some of them were repeated and reemphasized in Section III. 

Further repetition seems unnecessary her a. 
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SECTION V 



Confirmation or Questioning of the Need 
for Program Modifications Which Have 
Been Proposed as a Result of 
Project Evaluation 

The bilingual program should continue with vertical expansion. 
The evaluation has confirmed that the program was as successful as could 
be expected for a first-year developing program. The specifics of the 
indicated successful achievements have been discussed in previous 
sections of this audit report. 

The consortium approach has been well coordinated and the program 
should continue as a cooperative effort between Harlandale, San Marcos, 
and Southwest Texas State University. 

The curriculum guides have been most useful but the need for 
revision is apparent to everyone. It is recommended that the revision 
not be limited to suggestions made by the classroom teachers but that 
their suggestions should certainly be given careful consideration. 

The need for more trained bilingual teachers who are closely 
identified with the culture is apparent. Therefore, teacher training 
programs should be expanded and made more comprehensive. Present 
accomplishments are to be commended. An evaluator should be added to 
the project so that these duties would not have to be perfoirmed by the 
project director. An evaluator could make the evaluation more compre- 
hensive, develop and use more instruments, supervise test administra- 
tion more closely, extract more information from the data by using 
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additional statistical processes, and do more Interpretative evalua- 
tion because he would have more time* 

Progress could be more adequately measured by having as much time 
elapse between pre- and post-tests as possible. 

Adequate Spanish language tests are badly needed* 

Dr. B, E. Hughes, the program director, and Mrs. Dorcas DeSoto, 
the Harlandale Coordinator, are to be commended for their capable and 
enthusiastic leadership. The new coordinator in San Marcos should be 
equally competent. 

The auditor wishes to express his appreciation for the many 
courtesies and willing cooperation extended to him by everyone con- 
nected with the program. 
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